{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    }
   },
   "source": [
    "### TAO remote client (Simple object detection training with YOLO-V4)\n",
    "\n",
    "Transfer learning is the process of transferring learned features from one application to another. It is a commonly used training technique where you use a model trained on one task and re-train to use it on a different task. Train Adapt Optimize (TAO) Toolkit  is a simple and easy-to-use Python based AI toolkit for taking purpose-built AI models and customizing them with users' own data.\n",
    "\n",
    "![image](https://d29g4g2dyqv443.cloudfront.net/sites/default/files/akamai/TAO/tlt-tao-toolkit-bring-your-own-model-diagram.png)\n",
    "\n",
    "\n",
    "### The workflow in a nutshell\n",
    "\n",
    "- Creating a dataset\n",
    "- Upload kitti dataset to the service\n",
    "- Running dataset convert\n",
    "- Getting a PTM from NGC\n",
    "- Model Actions\n",
    "    - Train\n",
    "    - Evaluate\n",
    "\n",
    "### Table of contents\n",
    "\n",
    "1. [Install TAO remote client ](#head-1)\n",
    "1. [Set the remote service base URL](#head-2)\n",
    "1. [Access the shared volume](#head-3)\n",
    "1. [Create the datasets](#head-4)\n",
    "1. [List datasets](#head-5)\n",
    "1. [Provide and customize dataset convert specs](#head-6)\n",
    "1. [Run dataset convert](#head-7)\n",
    "1. [Create a model experiment](#head-8)\n",
    "1. [Find yolo pretrained model](#head-9)\n",
    "1. [Customize model metadata](#head-10)\n",
    "1. [Provide train specs](#head-11)\n",
    "1. [Run train](#head-12)\n",
    "1. [Provide evaluate specs](#head-13)\n",
    "1. [Run evaluate](#head-14)\n",
    "\n",
    "### Requirements\n",
    "Please find the server requirements [here](https://docs.nvidia.com/tao/tao-toolkit/text/tao_toolkit_api/api_setup.html#)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import glob\n",
    "import subprocess\n",
    "import getpass\n",
    "import uuid\n",
    "import json\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "namespace = 'default'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    }
   },
   "source": [
    "### Install TAO remote client <a class=\"anchor\" id=\"head-1\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# SKIP this step IF you have already installed the TAO-Client wheel.\n",
    "! pip3 install nvidia-transfer-learning-client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# View the version of the TAO-Client\n",
    "! nvtl --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FIXME\n",
    "\n",
    "1. Assign the ip_address and port_number in FIXME 1 and FIXME 2 ([info](https://docs.nvidia.com/tao/tao-toolkit/text/tao_toolkit_api/api_rest_api.html))\n",
    "1. Set NGC API key in FIXME 3\n",
    "1. Assign path of DATA_DIR in FIXME 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Set the remote service base URL <a class=\"anchor\" id=\"head-2\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Define the node_addr and port number\n",
    "node_addr = \"<ip_address>\" # FIXME1 example: 10.137.149.22\n",
    "node_port = \"<port_number>\" # FIXME2 example: 32334\n",
    "# In host machine, node ip_address and port number can be obtained as follows,\n",
    "# ip_address: hostname -i\n",
    "# port_number: kubectl get service ingress-nginx-controller -o jsonpath='{.spec.ports[0].nodePort}'\n",
    "%env BASE_URL=http://{node_addr}:{node_port}/{namespace}/api/v1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# FIXME: Set ngc_api_key valiable\n",
    "ngc_api_key = \"<ngc_api_key>\" # FIXME3 example: zZYtczM5amdtdDcwNjk0cnA2bGU2bXQ3bnQ6NmQ4NjNhMDItMTdmZS00Y2QxLWI2ZjktNmE5M2YxZTc0OGyM\n",
    "\n",
    "# Exchange NGC_API_KEY for JWT\n",
    "identity = json.loads(subprocess.getoutput(f'nvtl login --ngc-api-key {ngc_api_key}'))\n",
    "\n",
    "%env USER={identity['user_id']}\n",
    "%env TOKEN={identity['token']}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    }
   },
   "source": [
    "### Access the shared volume <a class=\"anchor\" id=\"head-3\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Get PVC ID\n",
    "pvc_id = subprocess.getoutput(f'kubectl get pvc tao-toolkit-api-pvc -n {namespace} -o jsonpath=\"{{.spec.volumeName}}\"')\n",
    "print(pvc_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Get NFS server info\n",
    "provisioner = json.loads(subprocess.getoutput(f'helm get values nfs-subdir-external-provisioner -o json'))\n",
    "nfs_server = provisioner['nfs']['server']\n",
    "nfs_path = provisioner['nfs']['path']\n",
    "print(nfs_server, nfs_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "user = getpass.getuser()\n",
    "home = os.path.expanduser('~')\n",
    "\n",
    "! echo \"Password for {user}\"\n",
    "password = getpass.getpass()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Mount shared volume \n",
    "! mkdir -p ~/shared\n",
    "\n",
    "command = \"apt-get -y install nfs-common >> /dev/null\"\n",
    "! echo {password} | sudo -S -k {command}\n",
    "\n",
    "command = f\"mount -t nfs {nfs_server}:{nfs_path}/{namespace}-tao-toolkit-api-pvc-{pvc_id} ~/shared\"\n",
    "! echo {password} | sudo -S -k {command} && echo DONE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    }
   },
   "source": [
    "### Create the datasets <a class=\"anchor\" id=\"head-4\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    }
   },
   "source": [
    "We will be using NVIDIA's synthetic dataset on warehouse images based on the `kitti object detection dataset` format in this example. To find more details about kitti, please visit [here](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The dataset follows this structure**\n",
    "```\n",
    "$DATA_DIR/train\n",
    "├── images\n",
    "│   ├── image_name_1.jpg\n",
    "│   ├── image_name_2.jpg\n",
    "|   ├── ...\n",
    "└── labels\n",
    "    ├── image_name_1.txt\n",
    "    ├── image_name_2.txt\n",
    "    ├── ...\n",
    "$DATA_DIR/val\n",
    "├── images\n",
    "│   ├── image_name_5.jpg\n",
    "│   ├── image_name_6.jpg\n",
    "|   ├── ...\n",
    "└── labels\n",
    "    ├── image_name_5.txt\n",
    "    ├── image_name_6.txt\n",
    "    ├── ...\n",
    "```\n",
    "The file name should be same for images and labels folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "DATA_DIR = \"tao_synthetic_data\" #FIXME4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 cp s3://tao-detection-synthetic-dataset-dev/tao_od_synthetic_train.tar.gz $DATA_DIR/\n",
    "!aws s3 cp s3://tao-detection-synthetic-dataset-dev/tao_od_synthetic_val.tar.gz $DATA_DIR/\n",
    "\n",
    "!mkdir -p $DATA_DIR/train/ && rm -rf $DATA_DIR/train/*\n",
    "!mkdir -p $DATA_DIR/val/ && rm -rf $DATA_DIR/val/*\n",
    "\n",
    "!tar -xzf $DATA_DIR/tao_od_synthetic_train.tar.gz -C $DATA_DIR/train/\n",
    "!tar -xzf $DATA_DIR/tao_od_synthetic_val.tar.gz -C $DATA_DIR/val/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "train_dataset_id = subprocess.getoutput(f\"nvtl yolo-v4 dataset-create --dataset_type object_detection --dataset_format kitti\")\n",
    "print(train_dataset_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! rsync -ah --info=progress2 $DATA_DIR/train/images ~/shared/users/{os.environ['USER']}/datasets/{train_dataset_id}/\n",
    "! rsync -ah --info=progress2 $DATA_DIR/train/labels ~/shared/users/{os.environ['USER']}/datasets/{train_dataset_id}/\n",
    "! echo DONE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "eval_dataset_id = subprocess.getoutput(f\"nvtl yolo-v4 dataset-create --dataset_type object_detection --dataset_format kitti\")\n",
    "print(eval_dataset_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! rsync -ah --info=progress2 $DATA_DIR/val/images ~/shared/users/{os.environ['USER']}/datasets/{eval_dataset_id}/\n",
    "! rsync -ah --info=progress2 $DATA_DIR/val/labels ~/shared/users/{os.environ['USER']}/datasets/{eval_dataset_id}/\n",
    "! echo DONE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "infer_dataset_id = subprocess.getoutput(f\"nvtl yolo-v4 dataset-create --dataset_type object_detection --dataset_format raw\")\n",
    "print(infer_dataset_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! rsync -ah --info=progress2 $DATA_DIR/val/images ~/shared/users/{os.environ['USER']}/datasets/{infer_dataset_id}/\n",
    "! echo DONE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    }
   },
   "source": [
    "### List datasets <a class=\"anchor\" id=\"head-5\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pattern = os.path.join(home, 'shared', 'users', os.environ['USER'], 'datasets', '*', 'metadata.json')\n",
    "\n",
    "datasets = []\n",
    "for metadata_path in glob.glob(pattern):\n",
    "    with open(metadata_path, 'r') as metadata_file:\n",
    "        datasets.append(json.load(metadata_file))\n",
    "\n",
    "print(json.dumps(datasets, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    }
   },
   "source": [
    "### Provide and customize dataset convert specs <a class=\"anchor\" id=\"head-6\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Default train dataset specs\n",
    "! nvtl yolo-v4 dataset-convert-defaults --id {train_dataset_id} --action convert | tee ~/shared/users/{os.environ['USER']}/datasets/{train_dataset_id}/specs/convert.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Customize train dataset specs\n",
    "specs_path = os.path.join(home, 'shared', 'users', os.environ['USER'], 'datasets', train_dataset_id, 'specs', 'convert.json')\n",
    "\n",
    "with open(specs_path , \"r\") as specs_file:\n",
    "    specs = json.load(specs_file)\n",
    "\n",
    "specs[\"kitti_config\"][\"image_extension\"] = \".jpg\" # Setting to the dataset's image_file extension type\n",
    "\n",
    "with open(specs_path, \"w\") as specs_file:\n",
    "    json.dump(specs, specs_file, indent=2)\n",
    "\n",
    "print(json.dumps(specs, indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Default eval dataset specs\n",
    "! nvtl yolo-v4 dataset-convert-defaults --id {eval_dataset_id} --action convert | tee ~/shared/users/{os.environ['USER']}/datasets/{eval_dataset_id}/specs/convert.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Customize eval dataset specs\n",
    "specs_path = os.path.join(home, 'shared', 'users', os.environ['USER'], 'datasets', eval_dataset_id, 'specs', 'convert.json')\n",
    "\n",
    "with open(specs_path , \"r\") as specs_file:\n",
    "    specs = json.load(specs_file)\n",
    "\n",
    "specs[\"kitti_config\"][\"image_extension\"] = \".jpg\" # Setting to the dataset's image_file extension type\n",
    "\n",
    "with open(specs_path, \"w\") as specs_file:\n",
    "    json.dump(specs, specs_file, indent=2)\n",
    "\n",
    "print(json.dumps(specs, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    }
   },
   "source": [
    "### Run dataset convert <a class=\"anchor\" id=\"head-7\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "train_convert_job_id = subprocess.getoutput(f\"nvtl yolo-v4 dataset-convert --id {train_dataset_id}  --action convert \")\n",
    "print(train_convert_job_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def my_tail(logs_dir, log_file):\n",
    "    %env LOG_FILE={logs_dir}/{log_file}\n",
    "    ! mkdir -p {logs_dir}\n",
    "    ! [ ! -f \"$LOG_FILE\" ] && touch $LOG_FILE && chmod 666 $LOG_FILE\n",
    "    ! tail -f -n +1 $LOG_FILE | while read LINE; do echo \"$LINE\"; [[ \"$LINE\" == \"EOF\" ]] && pkill -P $$ tail; done\n",
    "    \n",
    "# Check status (the file won't exist until the backend Toolkit container is running -- can take several minutes)\n",
    "logs_dir = os.path.join(home, 'shared', 'users', os.environ['USER'], 'datasets', train_dataset_id, 'logs')\n",
    "log_file = f\"{train_convert_job_id}.txt\"\n",
    "\n",
    "my_tail(logs_dir, log_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "eval_convert_job_id = subprocess.getoutput(f\"nvtl yolo-v4 dataset-convert --id {eval_dataset_id}  --action convert \")\n",
    "print(eval_convert_job_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Check status (the file won't exist until the backend Toolkit container is running -- can take several minutes)\n",
    "logs_dir = os.path.join(home, 'shared', 'users', os.environ['USER'], 'datasets', eval_dataset_id, 'logs')\n",
    "log_file = f\"{eval_convert_job_id}.txt\"\n",
    "\n",
    "my_tail(logs_dir, log_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    }
   },
   "source": [
    "### Create a model experiment <a class=\"anchor\" id=\"head-8\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "network_arch = \"yolo_v4\"\n",
    "model_id = subprocess.getoutput(f\"nvtl yolo-v4 model-create --network_arch {network_arch} --encryption_key tlt_encode \")\n",
    "print(model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    }
   },
   "source": [
    "### Find yolo pretrained model <a class=\"anchor\" id=\"head-9\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pattern = os.path.join(home, 'shared', 'users', '*', 'models', '*', 'metadata.json')\n",
    "\n",
    "ptm_id = None\n",
    "for metadata_path in glob.glob(pattern):\n",
    "  with open(metadata_path, 'r') as metadata_file:\n",
    "    metadata = json.load(metadata_file)\n",
    "    ngc_path = metadata.get(\"ngc_path\")\n",
    "    metadata_architecture = metadata.get(\"network_arch\")\n",
    "    if metadata_architecture == network_arch and \"pretrained_object_detection:resnet18\" in ngc_path:\n",
    "      ptm_id = metadata[\"id\"]\n",
    "      break\n",
    "\n",
    "print(ptm_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    }
   },
   "source": [
    "### Customize model metadata <a class=\"anchor\" id=\"head-10\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "metadata_path = os.path.join(home, 'shared', 'users', os.environ['USER'], 'models', model_id, 'metadata.json')\n",
    "\n",
    "with open(metadata_path , \"r\") as metadata_file:\n",
    "    metadata = json.load(metadata_file)\n",
    "\n",
    "metadata[\"train_datasets\"] = [train_dataset_id]\n",
    "metadata[\"eval_dataset\"] = eval_dataset_id\n",
    "metadata[\"inference_dataset\"] = infer_dataset_id\n",
    "metadata[\"ptm\"] = ptm_id\n",
    "\n",
    "with open(metadata_path, \"w\") as metadata_file:\n",
    "    json.dump(metadata, metadata_file, indent=2)\n",
    "\n",
    "print(json.dumps(metadata, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    }
   },
   "source": [
    "### Provide train specs <a class=\"anchor\" id=\"head-11\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Default train model specs\n",
    "! nvtl yolo-v4 model-train-defaults --id {model_id} | tee ~/shared/users/{os.environ['USER']}/models/{model_id}/specs/train.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Customize train model specs\n",
    "specs_path = os.path.join(home, 'shared', 'users', os.environ['USER'], 'models', model_id, 'specs', 'train.json')\n",
    "\n",
    "with open(specs_path , \"r\") as specs_file:\n",
    "    specs = json.load(specs_file)\n",
    "\n",
    "specs[\"training_config\"][\"num_epochs\"] = 100\n",
    "specs[\"dataset_config\"][\"image_extension\"] = \"jpg\" # Setting to the dataset's image_file extension type\n",
    "\n",
    "specs[\"augmentation_config\"][\"output_width\"] = 1280 # Setting to the dataset's original resolution's width\n",
    "specs[\"augmentation_config\"][\"output_height\"] = 736 # Setting to the dataset's original resolution's height\n",
    "\n",
    "with open(specs_path, \"w\") as specs_file:\n",
    "    json.dump(specs, specs_file, indent=2)\n",
    "\n",
    "print(json.dumps(specs, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    }
   },
   "source": [
    "### Run train <a class=\"anchor\" id=\"head-12\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_job_id = subprocess.getoutput(f\"nvtl yolo-v4 model-train --id {model_id}\")\n",
    "print(train_job_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Check status (the file won't exist until the backend Toolkit container is running -- can take several minutes)\n",
    "logs_dir = os.path.join(home, 'shared', 'users', os.environ['USER'], 'models', model_id, 'logs')\n",
    "log_file = f\"{train_job_id}.txt\"\n",
    "\n",
    "my_tail(logs_dir, log_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Provide evaluate specs <a class=\"anchor\" id=\"head-13\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Default evaluate model specs\n",
    "! nvtl yolo-v4 model-evaluate-defaults --id {model_id} | tee ~/shared/users/{os.environ['USER']}/models/{model_id}/specs/evaluate.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Customize evaluate model specs\n",
    "specs_path = os.path.join(home, 'shared', 'users', os.environ['USER'], 'models', model_id, 'specs', 'evaluate.json')\n",
    "\n",
    "with open(specs_path , \"r\") as specs_file:\n",
    "    specs = json.load(specs_file)\n",
    "\n",
    "specs[\"dataset_config\"][\"image_extension\"] = \"jpg\" # Setting to the dataset's image_file extension type\n",
    "\n",
    "specs[\"augmentation_config\"][\"output_width\"] = 1280 # Setting to the dataset's original resolution's width\n",
    "specs[\"augmentation_config\"][\"output_height\"] = 736 # Setting to the dataset's original resolution's height\n",
    "\n",
    "with open(specs_path, \"w\") as specs_file:\n",
    "    json.dump(specs, specs_file, indent=2)\n",
    "\n",
    "print(json.dumps(specs, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run evaluate <a class=\"anchor\" id=\"head-14\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "eval_job_id = subprocess.getoutput(f\"nvtl yolo-v4 model-evaluate --id {model_id} --job {train_job_id}\")\n",
    "print(eval_job_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Check status (the file won't exist until the backend Toolkit container is running -- can take several minutes)\n",
    "log_file = f\"{eval_job_id}.txt\"\n",
    "my_tail(logs_dir, log_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Copy these 2 UUID's for FIXME4 and FIXME5 of yolo_optimization.ipynb\n",
    "print(model_id)\n",
    "print(train_job_id)"
   ]
  }
 ],
 "metadata": {
  "datalore": {
   "base_environment": "default",
   "computation_mode": "JUPYTER",
   "package_manager": "pip",
   "packages": [],
   "version": 1
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
